Overview

This PR implements a complete, production-ready Python package tablediff-arrow for fast, file-based diffs of Parquet/CSV/Arrow data. The package supports local and S3 files, provides keyed comparisons with numeric tolerances, and generates HTML/CSV reports—all built on Apache Arrow for high performance.

What's Included

Core Library (src/tablediff_arrow/)

Implemented a full-featured data comparison library with four main modules:

• loader.py - Handles loading Parquet, CSV, and Arrow IPC files from local filesystem or S3
• compare.py - Performs keyed table comparisons with configurable absolute and relative numeric tolerances
• reports.py - Generates styled HTML reports and structured CSV reports showing differences
• cli.py - Provides a command-line interface with rich options for file comparison

Key Features

Fast Performance: Built on Apache Arrow and PyArrow for efficient data processing of large datasets.

Multiple Format Support: Automatically detects and loads Parquet (.parquet, .pq), CSV (.csv), and Arrow IPC (.arrow, .feather) files.

S3 Integration: Optional S3 support via s3fs allows comparing files directly from S3 buckets without downloading.

Flexible Comparisons:

• Use single or multiple key columns for joins
• Configure absolute tolerances per column (e.g., amount:0.01 for ±$0.01)
• Configure relative tolerances per column (e.g., rate:0.001 for ±0.1%)

Rich Reports:

• Styled HTML reports with color-coded differences and interactive tables
• CSV reports with separate files for changes, left-only rows, right-only rows, and summary statistics

Command-Line Interface

# Basic comparison
tablediff left.parquet right.parquet -k id

# With numeric tolerance
tablediff data1.csv data2.csv -k id -t amount:0.01 -o report.html

# S3 files
tablediff s3://bucket/left.parquet s3://bucket/right.parquet -k id --s3

Python API

from tablediff_arrow import TableDiff

differ = TableDiff(
    key_columns=['id'],
    tolerance={'amount': 0.01},
    relative_tolerance={'price': 0.001}
)

result = differ.compare_files('left.parquet', 'right.parquet')
print(result.summary())

Testing & Quality Assurance

Comprehensive Test Suite: 24 tests covering all modules with 86% code coverage

• 7 tests for data loading (Parquet, CSV, Arrow formats)
• 8 tests for comparison logic (tolerances, multiple keys, edge cases)
• 3 tests for report generation (HTML, CSV)
• 6 tests for CLI functionality

Code Quality: All code passes strict quality checks

• Formatted with Black
• Linted with Ruff
• Type-checked with Mypy
• Pre-commit hooks configured for automated checking

CI/CD Infrastructure

GitHub Actions Workflow (.github/workflows/ci.yml):

• Automated testing on Ubuntu, macOS, and Windows
• Tests across Python 3.10, 3.11, and 3.12
• Runs linting, formatting, type checking, and tests
• Builds and validates the package

Pre-commit Configuration (.pre-commit-config.yaml):

• Automated code formatting and linting
• YAML/JSON/TOML validation
• Trailing whitespace and end-of-file fixes

Documentation

Comprehensive documentation included:

• README.md - Complete user guide with installation, usage examples, API documentation, and CLI reference
• QUICKSTART.md - 5-minute getting started guide with common use cases
• CONTRIBUTING.md - Developer guidelines for contributing to the project
• CHANGELOG.md - Version history following Keep a Changelog format
• examples.py - 6 working examples demonstrating all package features

Package Configuration

Modern Python packaging with pyproject.toml:

• Supports Python 3.10+
• Declares all dependencies (core and optional)
• Configures build system (setuptools)
• Sets up entry point for tablediff CLI command
• Configures development tools (Black, Ruff, Mypy, Pytest)

Verification

All functionality has been tested and verified:

• ✅ Package installs correctly in development mode
• ✅ All 24 tests pass with high coverage
• ✅ CLI works with real-world data
• ✅ Reports generate correctly (HTML and CSV)
• ✅ Examples run successfully
• ✅ Code quality checks pass

The package is production-ready and follows Python best practices for packaging, testing, documentation, and CI/CD.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.